Directional sound modification

ABSTRACT

Various embodiments are directed towards an approach to modifying sound within an environment. In one aspect, a system includes one or more audio sensors coupled to a processor and arranged to detect sounds within the environment and further includes one or more audio output devices coupled to the processor. The processor executes instructions and, in response, operates to generate an audio signal that comprises an inverted copy of one or more sounds received from one or more directions, transmit the audio signal to the one or more audio output devices. The one or more audio devices combine the audio signal with the one or more sounds received from the one or more directions to produce modified sound.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of the co-pending U.S. patentapplication titled, “DIRECTIONAL SOUND MODIFICATION,” filed on Dec. 8,2014 and having Ser. No. 14/563,458. The subject matter of this relatedapplication is hereby incorporated herein by reference.

BACKGROUND

Field of the Disclosure

Embodiments disclosed herein relate to sound modification, and inparticular, generating audio signals to produce desired soundmodification for one or more selected directions within an environment.

Description of the Related Art

Individuals conventionally wear noise-cancelling or noise-attenuatingheadphones in busy or noisy environments in order to work withoutdistraction. Common types of headphones include in-ear headphones (or“ear buds”), on-ear headphones, and over-the-ear headphones. In manycases, the headphones generally provide a degree of passive noiseattenuation merely by being disposed over the ear canal of the wearer.Additionally, some headphones can provide active noise attenuation bygenerating sound waves that oppose sounds from the environment that aresensed by the headphones. Such headphones are typically configured toattenuate environmental noises falling within all or a selected portionof the audible frequency spectrum.

SUMMARY

In one embodiment, a sound modification system is disclosed thatincludes one or more audio sensors arranged to detect sounds within anenvironment and one or more audio output devices. The sound modificationsystem further includes a processing device coupled with the audiosensors and audio output devices, wherein the processing device operatesto generate an audio signal based on sounds detected from within one ormore selected directions within the environment, and to output thegenerated audio signal using the one or more audio output devices.

In another embodiment, a method for directional sound modification isdisclosed that includes selecting one or more directions within anenvironment to perform sound modification and detecting sounds fromwithin the one or more selected directions using one or more audiosensors coupled with a processing device. The method further includesusing the processing device to generate an audio signal based on thedetected sounds, and outputting the generated audio signal using one ormore audio output devices coupled with the processing device. The outputgenerated audio signal combines with the detected sounds to produce adesired sound modification.

In another embodiment, a computer program product for directional soundmodification is disclosed. The computer program product includes acomputer-readable device having computer-readable program code embodiedtherewith, where the computer-readable program code configured to selectone or more directions within an environment to perform soundmodification, to detect sounds from within the one or more selecteddirections using one or more audio sensors coupled with a processingdevice, to generate, using the processing device, an audio signal basedon the detected sounds, and to output the generated audio signal usingone or more audio output devices coupled with the processing device. Theoutput generated audio signal combines with the detected sounds toproduce a desired sound modification.

Other systems, methods, features, and advantages of the disclosure willbe, or will become, apparent to one of skill in the art upon examinationof the following figures and detailed description. It is intended thatall such additional systems, methods, features and advantages beincluded within this description, be within the scope of the disclosure,and be protected by the following claims.

BRIEF DESCRIPTION OF THE DRAWINGS

So that the manner in which the above recited features of the presentdisclosure can be understood in detail, a more particular description ofthe disclosure, briefly summarized above, may be had by reference toembodiments, some of which are illustrated in the appended drawings. Itis to be noted, however, that the appended drawings illustrate onlytypical embodiments of this disclosure and are therefore not to beconsidered limiting of its scope, for the disclosure may admit to otherequally effective embodiments.

FIGS. 1A and AB illustrate sound modification systems, according tovarious embodiments.

FIGS. 2A-2F illustrate sound modification systems deployed withindifferent environments, according to various embodiments.

FIG. 3 illustrates selection of directions for sound modification withinan environment, according to one embodiment.

FIG. 4 illustrates operation of a sound modification system in a noisyenvironment, according to one embodiment.

FIG. 5 illustrates updating selected directions for sound modificationaccording to one embodiment.

FIG. 6 illustrates a method for directional sound modification,according to one embodiment.

DETAILED DESCRIPTION

In the following description, numerous specific details are set forth toprovide a more thorough understanding of the present disclosure.However, it will be apparent to one of skill in the art that the presentdisclosure may be practiced without one or more of these specificdetails.

Embodiments disclosed herein include a sound modification system thatincludes one or more audio sensors arranged to detect sounds within anenvironment and one or more audio output devices. The sound modificationsystem further includes a processing device coupled with the audiosensors and audio output devices, wherein the processing devicesoperates to generate an audio signal based on sounds detected fromwithin one or more selected directions within the environment, and tooutput the generated audio signal using the one or more audio outputdevices. The output generated audio signal combines with the detectedsounds to produce a desired sound modification.

The sound modification system may be implemented in various forms ofaudio-based systems, such as personal headphones, home stereo systems,car stereo systems, etc. The sound modification system may selectivelyprovide noise attenuation, amplification, or any other desired audioeffects for modifying detected sounds. The sound modification system mayperform its processing functions using a dedicated processing deviceand/or a separate computing device such as a user's mobile computingdevice. The sound modification system may detect sounds from theenvironment using any number of audio sensors, which may be attached toor integrated with other system components or disposed separately. Thedetected sounds and selected directions may be used to generate atwo-dimensional (2D) or three-dimensional (3D) map of the environment,and the processing device may update the map based on changes to userorientation, as well as changes in relative distance between a user andvarious noise sources.

FIGS. 1A and 1B illustrate sound modification systems, according tovarious embodiments. As shown, sound modification system 100 includes aprocessing device 110, memory 120, input/output (I/O) 130, input device140, audio sensors 150, and audio output devices 155. The processingdevice 110 may include any processing element capable of performing thefunctions described herein. While depicted as a single element withinsound modification system 100, processing device 110 is intended torepresent a single processor, multiple processors, a processor orprocessors having multiple cores, as well as combinations thereof.Memory 120 may include a variety of computer readable media selected fortheir size, relative performance, or other capabilities: volatile and/ornon-volatile media, removable and/or non-removable media, etc. Memory120 may include cache, random access memory (RAM), storage, etc. Memory120 may include one or more discrete memory modules, such as dynamic RAM(DRAM) dual inline memory modules (DIMMs). Of course, various memorychips, bandwidths, and form factors may alternately be selected. Storageincluded as part of memory 120 may typically provide a non-volatilememory for the sound modification system 100, and may include one ormore different storage elements such as Flash memory, a hard disk drive,a solid state drive, an optical storage device, and/or a magneticstorage device.

Memory 120 may include one or more modules for performing functionsdescribed herein. As shown, memory 120 includes an audio signal module122 for generating audio signals to provide desired sound modificationsfor various selected directions, and an environmental map module 124 forcreating a 2-dimensional (2D) or 3-dimensional (3D) mapping of noisesources within the environment. Audio signal module 122 may generallyproduce audio signals in the form of a scaled and possibly inverted copyof detected sounds, but may also generate other waveforms in order toproduce the desired sound modification. For example, the audio signalmodule 122 might generate periodic audio signals or even random noise.The environmental map module 124 may separately include noise data 126that reflects input from audio sensors 150, direction data 128 thatreflects directions for sound modification (whether originally selecteddirections or updated) within the environment, and orientation data 129that reflects the relative orientation of at least one of the audiosensors 150, audio output devices 155, and a user of the soundmodification system 100.

The processing device 110 may communicate with other devices, such asperipheral devices or other networked computing devices, usinginput/output (I/O) 130. I/O 130 may include any number of different I/Oadapters or interfaces used to provide the functions described herein.I/O 130 may include wired and/or wireless connections, and may usevarious formats or protocols. In one example, the processing device 110through I/O 130 may determine selected directions for sound modificationusing input devices 140 that are connected using a wireless connection,such as Bluetooth® (a registered trademark of the Bluetooth SpecialInterest Group) or Wi-Fi® (a registered trademark of the Wi-FiAlliance), may detect environmental sounds using audio sensors 150 overwired connections, and may provide appropriate audio signals to audiooutput devices 155 over a separate wired or wireless connection toproduce a desired sound modification for the detected sounds in theselected directions.

I/O 130 may also include network interfaces that couple the processingdevice 110 to one or more networked computing devices through a network160. Examples of networked computing devices include a server, a desktopcomputer, a mobile computing device such as a smartphone or tabletcomputer, and a worn device such as a watch or headphones or ahead-mounted display device. Of course, other types of computing devicesmay also be networked with processing device 110. Network 160 mayinclude one or more networks of various types, including a local area orlocal access network (LAN), a general wide area network (WAN), and/or apublic network (e.g., the Internet). In some embodiments, the networkedcomputing devices may be used as input devices 140, audio sensors 150,and/or audio output devices 155.

Input devices 140 are coupled with the processing device 110 and providevarious inputs to the processing device 110 for performing directionalsound modification. As shown, input devices 140 include sensor devices142 and an orientation device 144. Sensor devices 142 may be provided tocapture input from users of the sound modification system 100, and mayinclude one or more types of sensors. For example, a user's input toselect directions for sound modification may include gestures, such asvarious movements or orientations of the hands, arms, eyes, or otherparts of the body. To detect user's input, sensor devices 142 mayinclude visual sensors such as infrared (IR) sensors, thermal sensors,and/or imaging devices such as a charge-coupled device (CCD) orcomplementary metal-oxide-semiconductor (CMOS) sensor device. Sensordevices 142 may also include inertial sensors, such as a gyroscope oraccelerometer. Sensor devices 142 may be worn or carried by the user, ormay be disposed separately (i.e., existing as, or included with, aseparate device). Of course, other types of sensor devices may also beincluded in sensor devices 142. to perform the various functions ofreceiving user input, which may include capacitive sensors, infraredsensors, magnetic sensors, sonar sensors, radar sensors, lidar sensors,neural sensors, and so forth.

In some embodiments, input devices 140 may include a user interface toreceive user selection of directions for sound modification. The userinterface may take any feasible form for providing the functionsdescribed herein, such as one or more buttons, toggles, sliders, dials,knobs, etc., or as a graphical user interface (GUI). The GUI may beprovided through any component of the sound modification system 100. Inone embodiment, the GUI may be provided by a separate computing devicethat is communicatively coupled with the processing device 110, such asthrough an application running on a user's mobile or wearable computingdevice. To provide preferential selection of sound modification, theuser interface may allow user input for various parameters such asdirection(s), type, and amount of sound modification to be performed.The parameters may be updated by the user or may be automaticallyupdated during operation.

In another example, the user interface may receive verbal commands forselecting directions and other sound modification parameters. In thiscase, input devices 140 may include one or more audio sensors, which maybe different or the same as the audio sensors 150. The processing device110 may perform speech recognition on the received verbal commandsand/or compare the verbal commands against commands stored in memory120. After verifying the received verbal commands, the processing device110 may carry out the commanded function for the sound modificationsystem (for example, altering sound modification parameters to specifiedlevels).

Orientation device 144 provides information about the orientation of theaudio sensors, audio output devices, and/or a user relative to theenvironment (and more specifically, relative to noise sources within theenvironment). The orientation device may provide two-dimensional (2D) orthree-dimensional (3D) orientation data to the processing device 110,which may integrate the orientation data into maps of the noiseenvironment. Orientation device 144 may include one or more sensordevices capable of detecting user orientation, such as a magnetometer,gyroscope, accelerometer, or imaging device. Orientation device 144 maybe worn by the user or may be disposed separately.

Audio sensors 150 are included to capture sounds occurring in theenvironment. The captured sounds may be used by the processing device togenerate appropriate directional sound modification. The audio sensorsmay be a plurality of microphones or other transducers or sensorscapable of converting sound waves into an electrical signal. The audiosensors may include an array of sensors that includes sensors of asingle type, or a variety of different sensors. Audio sensors 150 may beworn by a user, or disposed separately at a fixed location or movable.The audio sensors may be disposed in any feasible manner in theenvironment. In several embodiments, the audio sensors 150 are generallyoriented outward relative to audio output devices 155, which aregenerally disposed inward of the audio sensors 150 and also orientedinward. Such an orientation may be particularly beneficial for isolatingone or more regions for which sound modification is to be performed(i.e., using output from the audio output devices 155) from the rest ofthe environment. In one example, the audio sensors 150 may be orientedradially outward from a user, while the audio output devices 155 areoriented radially inward toward the user.

Audio output devices 155 are included to output generated audio signalsto provide appropriate sound modification corresponding to one or moreselected directions within the environment. Of course, the soundmodification audio signals may be simultaneously driven on the audiooutput devices 155 with other audio signals (e.g., music or other audioplayback). The audio output devices may use conventional audio outputtechniques, such as loudspeakers or other suitable electroacousticdevices. Audio output devices 155 may be implemented using any number ofdifferent conventional form factors, such as discrete loudspeakerdevices, around-the-ear (circumaural), on-ear (supraaural), or in-earheadphones, hearing aids, wired or wireless headsets, body-worn (head,shoulder, arm, etc.) listening devices, body-worn close-rangedirectional speakers or speaker arrays, body-worn ultrasonic speakerarrays, and so forth. The audio output devices 155 may be worn by a useror disposed separately at a fixed location or movable. As discussedabove, the audio output devices 155 may be disposed inward of the audiosensors 150 and oriented inward toward a particular region or user.

FIG. 1A shows one embodiment in which various components of the soundmodification system 100 may be distributed across several devices. FIG.1B shows another embodiment in which computing components (i.e.,processing device 110, memory 120, and I/O 130) of sound modificationsystem 170 are included in a discrete computing device 180. Generally,the computing device 180 receives input from the one or more inputdevices 140 and audio sensors 150, generates the audio signals fordirectional sound modification, and outputs the generated audio signalsusing audio output devices 155. As will be seen below, computing device180 may be disposed in relative proximity to the audio sensors 150 andaudio output devices 155.

FIGS. 2A-2.F illustrate sound modification systems deployed withindifferent environments, according to various embodiments. FIG. 24illustrates the sound modification system implemented in over-the-earheadphones 200, according to one embodiment. The headphones 200 includeear cups 205 that are provided to comfortably interface with a user'shead and to cover the user's ears. The headphones 200 also include ahousing 210 that connects to each ear cup 205, providing support to theear cups, the speaker elements, as well as any other components includedin the headphones 200. As shown, the headphones 200 include a processingmodule 211, a plurality of microphones 212, one or more buttons 213, afeedback device 214, and a power source 215. Of course, the person ofordinary skill in the art will recognize that other components, thoughnot explicitly mentioned here, may also be included in headphones 200.

At a minimum, processing module 211 includes ability to receive audiosignals through a wired or a wireless connection and to output the audiosignal to the speaker elements of the headphones 200. Processing module211 may also include one or more active or passive devices formodulating the received audio signals. Processing module 211 may includethe processing device 110 along with other functionality described abovewith respect to the sound modification systems 100, 170 (e.g., sensordevices 142, orientation device 144) to provide directional soundmodification within the environment. In one embodiment, the processingmodule 211 may be the computing device 180. Additionally, processingmodule 211 may be coupled with one or more separate computing devicesthat provide the sound modification audio signals, and optionallyprovide media to be output to the speaker elements of the headphones200. The computing device may be a mobile or worn computing device ofthe user, such as a laptop, smartphone, tablet, smartwatch, etc.

The microphones 212 may be used as the audio sensors 150 andpreferentially disposed in a particular arrangement. For example, anarray of microphones 212 may be distributed along the width of thehousing 210 and oriented outward to capture noise occurring in theenvironment outside the worn headphones. In one example, the microphonesmay be oriented radially outward, by following a curved outer surface ofthe housing 210 and/or by being individually oriented. Of course, themicrophones may be preferentially distributed along one or moredimensions or surfaces to provide a sound-capturing panorama of adesired shape and size. In one embodiment, the array of microphones mayinclude one or more microelectromechanical systems (MEMS) devices, eachMEMS device including a plurality of smaller transducers. The pluralityof transducers may be spatially separated so that the directionality ofthe sound events can be determined through arrival timing differences.The signals received from the transducers may then be processed andexamined for intensity, spectral, and timing cues to allow localizationof sound sources.

The one or more buttons 213 may he used as an input device 140 forselecting one or more directions within the environment for performingsound modification. The buttons 213 may be disposed on the housing 210,such as one or more buttons on the portions of housing 210 connected toeach ear cup 205. The buttons 213 may be disposed similarly to themicrophones 212 with each button corresponding specifically to one ormore of the microphones 212. In one embodiment, the buttons andmicrophones correspond in a 1:1 ratio. For example, pressing a buttonmay toggle whether or not sound modification is being performed on thesounds detected by the corresponding one or more microphones, or maychange sound modification settings (e.g., change the amount ofamplification or attenuation). In one embodiment, the buttons 213 may beprovided to cycle through a plurality of predetermined settings forsound modification, whether set by default or user-specified. In oneembodiment, the buttons 213 may be used as a trigger device for otherinputs. For example, the user may press a button and subsequently inputa verbal command or make a particular input gesture to select directionsor other sound modification parameters.

Feedback device 214 may be included to provide visual or haptic feedbackto a user. For example, feedback device 214 may include one or morelight emitting diodes (LEDs) or vibrating motors. In one embodiment, theLEDs may he disposed similarly to the microphones 212 and/or buttons213, and may indicate the selected directions for performing soundmodification. The feedback device may also acknowledge a successful userselection, e.g., by blinking or vibrating.

Power source 215 may be coupled with the processing module 211 andfeedback device 214 to provide power to each component. Power source 215may include replaceable or rechargeable batteries or other energystorage devices. Power source 215 may also include a connection to wallpower for powering the components and/or recharging the batteries.

FIG. 2B illustrates an example environment 220, in which theover-the-ear headphones 200 are worn by a user 225, according to oneembodiment. Based on intrinsic properties of the microphones and theirrelative dispositions within headphones 200, the various microphones areeach capable of sensing a minimum threshold level of sound, which maycorrespond to a particular distance 230 from the microphone. Incombination, the composite sensing regions of the various microphonesmay form an audio sensing zone 235, which is represented by a spatialarea or volume extending from the microphones into the ambientenvironment. The audio sensing zone 235 may have various shapes and/orsizes depending on the number, positioning, and orientation of themicrophones, as well as each microphone's capability (e.g., sensitivity,frequency response, etc.). In the simplified example depicted here,audio sensing zone 235 is represented by a sphere surrounding the headof user 225. Of course, more complex shapes are possible and expected,such as elongated shapes, shapes that include overlapping areas ofmicrophone coverage, or non-continuous shapes in which the microphonesdo not provide complete sound coverage. For any given device, such asheadphones 200, the device may have differing audio sensing zones atdifferent noise frequencies as the frequency-dependent properties ofeach microphone may be different.

As described here, the outer spatial limits of the audio sensing zonerepresent some predetermined minimum sound level (e.g., 3 decibels ordB). Of course, this does not require that a particular noise source bephysically located within the space defined by the audio sensing zone,but only that the noise source generates sufficient power to meet orexceed the threshold sound level at the outer limit.

FIG. 2C illustrates another example environment 240 for a soundmodification system, according to one embodiment. In this case, thesound modification system may be deployed in a home stereo system. Thehome stereo system may include a television 245 or other audiovisualdevice, a stereo receiver 247, and a plurality of speakers 250. Each ofthe speakers 250 may include drivers corresponding to differentfrequency ranges (e.g., tweeters, woofers, subwoofers) and may bepreferentially disposed within the environment 240 for audio quality.More specifically, the television 245 and speakers 250 may be disposedto provide optimal audio and video quality for one or more users at apredetermined location, e.g., seated on couch 242.

FIG. 2D illustrates a top view of environment 260, according to oneembodiment. In large part, the environment 260 is the same asenvironment 240, but environment 260 explicitly depicts audio sensorsand the corresponding audio sensing zone 277. One or more differenttypes of audio sensors may be included in the environment 260. The audiosensors may be attached to, or integrated with, various components ofthe home stereo system, such as audio sensors 255 disposed on thespeakers 250. Audio sensors may also be disposed separately, such asattached to a non-component of the home stereo system, or as astandalone sensor. Audio sensors 275 are attached to the exterior ofwalls 265 near windows 270, and may be used to modify outdoor noise(e.g., animals, neighbors, automotive/train/air traffic, etc.).Processing for the sound modification system may be natively performedby the stereo receiver 247, or may be performed by a separate computingdevice which is also able to output audio signals to the variousspeakers 250. The computing device could be a computing system includedwith the home stereo system, or alternately may be a mobile computingdevice of the user, such as a laptop, smartphone, tablet, smartwatch,etc.

FIG. 2E illustrates a sound modification system as implemented in anautomobile 280, according to one embodiment. As shown, automobile 280includes a passenger compartment 282, in which a plurality of speakers285 and an audio receiver 287 are located. The audio receiver 287 iscoupled with the speakers 285 and generally operates to receive an audioinput (AM/FM/satellite-based radio, compact disc, MP3 files, etc.) andto drive amplified and/or equalized audio signals to the speakers 285.The sound modification system may include a plurality of audio sensors290 disposed on the exterior of the automobile 280 and oriented outward.Though tour audio sensors are shown as disposed on the automobile'squarter panels, any number of sensors disposed in any interior orexterior location of the automobile are possible. In one embodiment,audio sensors may be disposed near the engine compartment 291 (such asbetween the engine compartment and the passenger compartment 282) inorder to preferentially modify engine sounds (e.g., attenuate oramplify). Processing for the sound modification system may be nativelyperformed by the audio receiver 287, or may be performed by a separatecomputing device which is also able to output audio signals to thevarious speakers 285. Again, the computing device could be a computingsystem included with the audio system, or alternately may be a mobilecomputing device of the user, such as a laptop, smartphone, tablet,smartwatch, etc.

FIG. 2F illustrates environment 292, in which the automobile 280 isoperating along road 295. As with other embodiments described above, thesensors 290 of the sound modification system correspond to an audiosensing zone 297. As environmental noises are detected by sensors 290,the sound modification system may generate audio signals to provide thedesired modification effect for sounds coming from selected directions.

FIG. 3 illustrates the selection of directions for sound modificationwithin an environment, according to one embodiment. Although oneparticular embodiment including headphones is depicted, the person ofordinary skill will understand that various alternative implementationsare also possible. Environment 300 provides a top-down depiction of auser 225 wearing headphones 200 on his or her head. The user 225 has aninitial orientation 305 within the environment 300. Though a simplified2D representation of the user orientation and the environment ispresented here, the person of ordinary skill will understand that thesame principles would also apply to a 3D representation e.g., capturingwhether the user is leaning head forward, back, to the left or rightside, etc.). An audio sensing zone 325 representing the compositesensing regions of the various microphones included with the headphones200 extends from the microphones into the ambient environment. Soundsdetected by headphones 200 as coming from pass-through area 310 withinthe audio sensing zone 325 are permitted to pass through to the userwithout applying an active sound modification. Sounds detected as comingfrom modification area 320 within the audio sensing zone 325, however,are combined with generated audio signals to produce a desired soundmodification.

A user may select the direction(s) for sound modification using anynumber of methods. In the simplified case shown in environment 300, theuser might select an entire side 320 to be attenuated or amplified(i.e., corresponding to one of the ear cups of the headphones 200).Alternatively, the user might specify an angle and angular width (say acenter angle of 90° from the current orientation 305, with a 180°width), or multiple angles (from 0°-180°).

As discussed above, the user may be able to provide this directionselection input through the use of pushbuttons, verbal commands,gestures, using a GUI, etc. In one embodiment, each side of headphones200 may include one or more buttons, so that user 225 may selectivelyapply sound modification for one or more directions merely by pressingcorresponding buttons. In another embodiment, the user may provideverbal commands for selecting the one or more directions, by selectingthe angles directly or indirectly (e.g., using words or phrases that arepre-mapped to certain angles). In another embodiment, the user nayprovide gestures in which the angles may be selected directly orindirectly. For example, a user could point to first and second anglesdefining the modification area 320, or could point at an object (e.g., aparticular noise source). In one embodiment, the orientation of theuser's eyes may be determined in conjunction with selecting the one ormore directions, so that by simply looking at a sound source thedirection may be determined. In this case, the direction may bedetermined based on the user's gaze after triggering the selection byspeaking a verbal command, pressing a button, etc. The soundmodification system may receive the user's input and set appropriateangles so that the object is completely included within modificationarea 320.

Along with selecting directions for sound modification, the user 225 mayalso specify the type and amount of modification (e.g., amplification,attenuation, and amounts of either). For example, a user might point toa noise source and say, “reduce this noise by 50%” or “reduce any noisesfrom this direction by 3 dB.” In another example, a user wearingheadphones who wants to be made aware when a coworker approaches his orher office might point to the open office door and say, “increase soundscoming from this direction by 35%.” The type and amount of soundmodification may vary for different modification areas. In addition todirections, a user may also specify that certain frequency ranges are tobe modified. The user may specify these by indicating specific frequencyvalues or by selecting pre-mapped frequency ranges (corresponding tospeech, automobile traffic, or other common noise source ranges). Themodification areas specified by the user (such as modification area 320)may track the user's orientation, or may remain fixed despite changes tothe user's orientation. For example, the user may select all sounds fromhis or her right side to be sound modified. If the correspondingmodification area is set to track the user, the sounds coining from theuser's right side at any instant (even if the user has moved) willcontinue to be sound modified.

In some embodiments, input from one or more sensors may be correlatedwith various sound sources to determine which sounds are most disruptivefor a user. The disruption determination may be based on a temporalcomparison of sensor measurements against various sounds in theenvironment. Example sensor measurements include brain activity todetermine a loss of focus or concentration e.g., using neural sensors)or detecting eye or head movement (e.g., a larger movement may generallycorrelate to a disruption). Based on the disruption determination, whenaudio sensors detect sounds that meet criteria sufficiently similar tothe disruptive sounds, directions for sound modification may bedetermined and applied automatically for these sounds.

As discussed above, the sound modification systems may generate mappingsof the environment to reflect detected noise and the one or moreselected directions for sound modification. The sound modificationsystems may transform the mappings according to the user's currentlocation and orientation before generating the audio signals for soundmodification. A 2D version of the map may be similar in appearance tothe depictions of FIG. 3. Whereas the modification areas of the 2D mapare generally represented as wedges projecting from the user (or fromone or more microphones), a 3D map might include various vectorsprojecting from the user or microphones, which in 3D space might beconical or appear cone-like.

As a part of generating the environmental maps, the sound modificationsystems may also estimate discrete noise source locations for thedetected sounds, and may plot those estimated locations in the maps. Themaps may use any known coordinate systems, such as Cartesian, polar, orspherical coordinates. These maps may further be linked to an absoluteposition of the user (provided via sensor devices, such as a GlobalPositioning System (GPS) sensor). When linked to an absolute position,the maps may be useful for other users of sound modification systems.For example, noise maps that are generated while a headphone-wearinguser walks down a busy road could be stored to a server and laterprovided to other users in that vicinity, which might decrease orprevent redundant processing by the various sound modification systems.

Environment 330 also provides a top-down depiction of the user 225wearing headphones 200 on his or her head. User 225 has same orientation305, but in this example wishes to specify different directions for themodification area 350 (in this case, the area is located behind theuser). The user 225 setting one or more modification areas 350 may alsooperate to define one or more pass-through areas 340 within the audiosensing zone 325. Again, the user may select the directions byspecifying particular angles. In an alternate embodiment, the user mayspecify a direction or particular angle, along with a modifier todescribe the relative width of the modification area 350 (e.g.,“narrow,” “moderate,” “wide”). The modifiers may be pre-mapped torepresent certain angular widths. In an alternate embodiment, the usermay specify one angle (e.g., 180 degrees from current orientation, or“behind me”) and a predetermined default angular width is applied tocreate the modification area 350. Of course, after initially setting themodification area 350, the user may select entirely new modificationareas or may make incremental adjustments to the modification area 350.For example, the user may identify the modification area and providespecific angle or angular width changes, or may specifywidening/narrowing the modification area and/or shifting themodification area relative to user orientation.

Environment 360 also provides a top-down depiction of the user 225wearing headphones 200 on his or her head. User 225 has same orientation305, but in this example wishes to specify directions for two differentmodification areas 380 ₁, 380 ₂. Setting the modification areas 380 ₁,380 ₂ may also operate to define one or more pass-through areas 370 ₁,370 ₂ within the audio sensing zone 325. The user may specify angles orranges of angles for each modification area 380 ₁, 380 ₂, which may beselected simultaneously or at different times. As before, user mayalternately use verbal descriptors to set the width of each modificationarea (for example, “a wide range centered at 135°, and a narrow range at315°”). Alternatively, the user may specify an angle, and apredetermined default angular width is applied.

FIG. 4 illustrates operation of a sound modification system deployed ina noisy environment, according to one embodiment. A multi-story officebuilding 405 is depicted, in which user 225 is wearing headphones 200while working at a workstation 410. A co-worker in an adjacentworkstation 420 is talking loudly on a phone, which may be distractingto the user. Meanwhile, roadwork 440 is occurring on the street 450outside the office building 405, also creating noise that may distractthe user.

Using any of the various techniques described above, the user may selectdirections for noise modification corresponding to these noise sources.In this case, the user may desire to attenuate the noise sources.Although not shown, user may additionally or alternatively select one ormore directions in which to enhance sound (e.g., amplify and/orequalize), such as from the directions of the user's computer or phone.After the user specifies the directions corresponding to the noisesources, the sound modification system may determine the loudest noisesource(s) within a predetermined range of the specified directions, asthe user may not have provided a precise indication of the directions,and as it is likely that the loudest noises coming from the selecteddirections are what the user seeks to modify. A 3D mapping of theenvironment 400 may thus include vectors 430 and 460 projecting from theuser 225 (or rather, the corresponding microphones included inheadphones 200). The vectors 430, 460 indicate that sound modificationwill be performed for sounds detected as coming from the correspondingdirections.

FIG. 5 illustrates updating selected directions for sound modification,according to one embodiment. In environment 500, a user 225 is depictedas wearing headphones 200 on his or her head, while in an initialorientation 505. Two point noise sources 510 ₁, 510 ₂ are included inthe environment 500. One noise source 510 ₁ is disposed within the audiosensing zone 525 of the headphones 200, while the other noise source 510₂ is disposed outside. Thus, sounds detected from directionscorresponding to pass-through areas 520 are not sound-modified, whilesounds from the modified areas 515 ₁, 515 ₂ are modified by the soundmodification system. The sound modification system may generate a map ofthe environment 500 based on these selected directions and noisesources.

In environment 530, the user has turned his or her entire body (orperhaps just his/her head), such that orientation of the user (or of theaudio sensors of headphones 200) changes from orientation 505 toorientation 535. In one embodiment, the sound modification system isconfigured to track the noise sources for changes in user orientation.Though the user has re-oriented, the noise sources 510 ₁, 510 ₂ remainin the same positions, and thus the modified areas 510 ₁, 510 ₂ remainstatic relative to the noise sources. No matter what changes to userorientation occur, the noise sources will continue to be sound modified.While the environments are shown in 2D for simplicity, the person ofordinary skill will understand that similar implementations may be madein 3D space.

In one embodiment, the sound modification system is configured to alsotrack displacement of the user and/or the noise sources. This may bedone in addition to tracking the noise sources for changes in userorientation. In environment 550, the user has an initial orientation555. Again, two point noise sources 510 ₁, 510 ₂ are included. Themodified area 515 ₁ corresponding to noise source 510 ₁ has an initialangular width α₁, and modified area 515 ₂ corresponding to noise source510 ₂ has an initial angular width β₁.

In environment 560. user maintains the same orientation 555 but arelative displacement occurs between the user and the two point noisesources 510 ₁, 510 ₂. For example, the user may be moving and/or one orboth of the noise sources may be moving. Modification area 515 ₁ hasshifted relative to the user orientation and is now modification area565 ₁, and has a smaller angle α₂ indicating an increase in distancebetween the user and the noise source 510 ₁. Modification area 515 ₂ hasalso shifted relative to the user orientation and is now modificationarea 565 ₂, but has an angle β₂ that is approximately the same size asangle β₁ (indicating that the distance between the user and noise sourceis approximately the same). Corresponding pass-through areas 570 fillthe remainder of the audio sensing zone 525.

FIG. 6 illustrates a method for directional sound modification,according to one embodiment. The method 600 may be used consistent withdescriptions of the various sound modification systems described above,and within the environments described in various embodiments. Method 600may be performed using a processing device of a sound modificationsystem, or using a separate computing device communicatively coupledwith the sound modification system, or using a combination of variousprocessing devices. For example, method 600 may be performed by anapplication executing on a user's mobile computing devicecommunicatively coupled with the sound modification system.

Method 600 begins at block 605, where one or more directions within theenvironment are selected for performing sound modification. Thedirections may be selected by a user, as well as the type and amount ofmodification to perform on sounds coming from the selected directions.The selected directions may be included in a 2D or 3D map generated ofthe noise environment, and may form one or more modification areas thatmay selectively track user orientation, user displacement, and/or noisesource displacement.

At block 610, the processing device determines whether sounds aredetected as coming from the one or more selected directions by one ormore audio sensors of the sound modification system. If no sounds aredetected, or if any detected sounds are determined to come frompass-through areas falling outside the selected directions, the methodproceeds to block 615 (“NO”) and any detected sounds from thenon-selected directions are permitted to pass through without providingactive sound modification. The method may generally loop through block610, whether continuously or at discrete time intervals, until detectedsounds correspond to the one or more selected directions (“YES”), whenthe method proceeds to block 625.

At block 625, the processing device generates an audio signal based onthe detected sounds corresponding to the one or more selecteddirections. The audio signal is also based on the desired soundmodification specified by the user, such as attenuation or amplificationof the detected sounds, and amounts of either. The audio signal maygenerally take the form of a scaled and possibly inverted copy of thedetected sounds, but other waveforms may be used to generate the desiredsound modification.

At block 635, the generated audio signal is output to produce thedesired sound modification. This may include driving selected ones ofthe audio output devices with the output signal (for example, audiooutput devices whose orientations are most closely aligned with theselected directions). The method may end following block 635, or mayreturn to block 610, whether as a continuous loop or at discrete timeintervals.

The present disclosure may be embodied in an apparatus, a system, amethod, and/or a computer program product. The computer program productmay include a computer readable storage medium (or media) havingcomputer readable program instructions thereon for causing a processorto carry out aspects of the present disclosure.

The computer readable storage medium can be a tangible device that canretain and store instructions for use by an instruction executiondevice. The computer readable storage medium may be, for example, but isnot limited to, an electronic storage device, a magnetic storage device,an optical storage device, an electromagnetic storage device, asemiconductor storage device, or any suitable combination of theforegoing. A non-exhaustive list of more specific examples of thecomputer readable storage medium includes the following: a portablecomputer diskette, a hard disk, a random access memory (RAM), aread-only memory (ROM), an erasable programmable read-only memory (EPROMor Flash memory), a static random access memory (SRAM), a portablecompact disc read-only memory (CD-ROM), a digital versatile disk (DVD),a memory stick, a floppy disk, a mechanically encoded device such aspunch-cards or raised structures in a groove having instructionsrecorded. thereon, and any suitable combination of the foregoing. Acomputer readable storage medium, as used herein, is not to be construedas being transitory signals per se, such as radio waves or other freelypropagating electromagnetic waves, electromagnetic waves propagatingthrough a waveguide or other transmission media (e.g., light pulsespassing through a fiber-optic cable), or electrical signals transmittedthrough a wire.

Computer readable program instructions described herein can bedownloaded to respective computing/processing devices from a computerreadable storage medium or to an external computer or external storagedevice via a network, for example, the Internet, a local area network, awide area network and/or a wireless network. The network may comprisecopper transmission cables, optical transmission fibers, wirelesstransmission, routers, firewalls, switches, gateway computers and/oredge servers. A network adapter card or network interface in eachcomputing/processing device receives computer readable programinstructions from the network and forwards the computer readable programinstructions for storage in a computer readable storage medium withinthe respective computing/processing device.

Computer readable program instructions for carrying out operations ofthe present disclosure may be assembler instructions,instruction-set-architecture (ISA) instructions, machine instructions,machine dependent instructions, microcode, firmware instructions,state-setting data, or either source code or object code written in anycombination of one or more programming languages, including an objectoriented programming language such as Smalltalk, C++ or the like, andconventional procedural programming languages, such as the “C”programming language or similar programming languages. The computerreadable program instructions may execute entirely on the user'scomputer, partly on the user's computer, as a stand-alone softwarepackage, partly on the user's computer and partly on a remote computeror entirely on the remote computer or server. In the latter scenario,the remote computer may be connected to the user's computer through anytype of network, including a local area network (LAN) or a wide areanetwork (WAN), or the connection may be made to an external computer(for example, through the Internet using an Internet Service Provider).In some embodiments, electronic circuitry including, for example,programmable logic circuitry, field-programmable gate arrays (FPGA), orprogrammable logic arrays (PLA) may execute the computer readableprogram instructions by utilizing state information of the computerreadable program instructions to personalize the electronic circuitry,in order to perform aspects of the present disclosure.

Aspects of the present disclosure are described herein with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems and computer program products according to embodiments of thedisclosure. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer readable program instructions.

These computer readable program instructions may be provided to aprocessor of a general purpose computer, special purpose computer, orother programmable data processing apparatus to produce a machine, suchthat the instructions, which execute via the processor of the computeror other programmable data processing apparatus, create means forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks. These computer readable program instructionsmay also be stored in a computer readable storage medium that can directa computer, a programmable data processing apparatus, and/or otherdevices to function in a particular manner, such that the computerreadable storage medium having instructions stored therein comprises anarticle of manufacture including instructions which implement aspects ofthe function/act specified in the flowchart and/or block diagram blockor blocks.

The computer readable program instructions may also be loaded onto acomputer, other programmable data processing apparatus, or other deviceto cause a series of operational steps to be performed on the computer,other programmable apparatus or other device to produce a computerimplemented process, such that the instructions which execute on thecomputer, other programmable apparatus, or other device implement thefunctions/acts specified in the flowchart and/or block diagram block orblocks.

The flowchart and block diagrams in the Figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods, and computer program products according to variousembodiments of the present disclosure. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof instructions, which comprises one or more executable instructions forimplementing the specified logical function(s). In some alternativeimplementations, the functions noted in the block may occur out of theorder noted in the figures. For example, two blocks shown in successionmay, in fact, be executed substantially concurrently, or the blocks maysometimes be executed in the reverse order, depending upon thefunctionality involved. it will also be noted that each block of theblock diagrams and/or flowchart illustration, and combinations of blocksin the block diagrams and/or flowchart illustration, can be implementedby special purpose hardware-based systems that perform the specifiedfunctions or acts or carry out combinations of special purpose hardwareand computer instructions.

While the foregoing is directed to embodiments of the presentdisclosure, other and further embodiments of the disclosure may bedevised without departing from the basic scope thereof, and the scopethereof is determined by the claims that follow.

What is claimed is:
 1. A system, comprising: one or more audio sensorscoupled to a processor and arranged to detect sounds within anenvironment; and one or more audio output devices coupled to theprocessor, wherein the processor executes instructions and, in response,operates to: generate an audio signal that comprises an inverted copy ofone or more sounds received from one or more directions; and transmitthe audio signal to the one or more audio output devices, wherein theone or more audio devices combine the audio signal with the one or moresounds received from the one or more directions to produce modifiedsound.
 2. The system of claim 1, wherein the modified sound results fromattenuating the one or more sounds received from the one or moredirections.
 3. The system of claim 1, wherein the inverted copy of thesounds received from the one or more directions is scaled.
 4. The systemof claim 1, further comprising a first input device, wherein the one ormore directions are selected via the first input device.
 5. The systemof claim 4, wherein the first input device comprises a sensor devicearranged to detect at least one of a gesture and a verbal commandprovided by a user.
 6. The system of claim 1, further comprising anorientation device that is coupled to the processor and operates todetermine an orientation of at least one of the one or more audiosensors within the environment and the one or more audio output deviceswithin the environment.
 7. The system of claim 6, wherein the one ormore directions are updated based on the orientation.
 8. The soundmodification system of claim 1, wherein the processor further operatesto generate a map of the environment that reflects the one or moredirections.
 9. The sound modification system of claim 8, wherein thegenerated map of the environment includes one or more identified soundsources, and wherein the processor is configured to update the map ofthe environment based a relative motion of at least one of the one ormore audio sensors and the one or more audio output devices, and the oneor more identified sound sources.
 10. A method for directional soundmodification, comprising: receiving direction input data from an inputdevice coupled to a processor; detecting sounds received from the one ormore directions based on the direction input data; generating, via theprocessor, an audio signal that comprises an inverted copy of the soundsreceived from the one or more selected directions; and transmitting theaudio signal to one or more audio output devices coupled to theprocessor to generate audio output, wherein the one or more audiodevices combine the audio signal with the one or more sounds receivedfrom the one or more directions to produce modified sound.
 11. Anon-transitory computer-readable storage medium storing instructionsthat, when executed by a processor, cause the processor to perform thesteps of: detecting sounds received from one or more directions withinan environment; generating an audio signal that comprises an invertedcopy of the sounds received from the one or more directions; andtransmitting the audio signal to one or more audio output devicescoupled to the processor to generate audio output, wherein the one ormore audio devices combine the audio signal with the one or more soundsreceived from the one or more directions to produce modified sound.